In a previous blog, Meet Red Hat Device Edge with MicroShift, we demonstrated how to build an x86 Red Hat Device Edge 8.7 image that included MicroShift 4.12. This blog will build upon those concepts but add a few new twists. First we are going to build the image using the Red Hat Enterprise 9.2 Beta. We will also be using an early candidate release of MicroShift 4.13.0-ec.4. Together we will build another Red Hat Device Edge image but this time instead of x86 based we will make an Arm based image. We will be doing the majority of our work on a Adlink Ampere Altra Developer Platform and leveraging Arm based Kvm virtual machines on the system as our image building host and our pseudo edge device. It should be noted this deployment is still in pre-release form and not yet officially supported.

Before we cover building and deploying the image let's briefly go over the environment. On the Adlink Ampere Altra Developer Platform KVM host we have 64 cores and 196GB of memory. We have also installed Red Hat Enterprise Linux 9.1. We will configure two KVM virtual machines on our Adlink Ampere Altra Developer Platform. These virtual machines have the same configuration as follows:

One of the virtual machines will have Red Hat Enterprise Linux 9.2 Beta installed on it just as a general server system where we will build our image. The other we will not touch until we have built our image which we will then deploy on that virtual machine. Given we need to build an image let's move onto that process.

To build our Red Hat Device Edge image we will login to the first virtual machine we have installed Red Hat Enterprise Linux 9.2 Beta on. We need to make sure that we have a few packages installed so that we have the capabilities to use ImageBuilder to build our Arm image. It should be noted that at this time ImageBuilder cannot cross build to different architectures which is why we are doing all of this on an Arm system. First make sure the following repos are available on the RHEL 9.2 Beta virtual machine:

Next we need to make sure we have the following packages installed on the host as these will all be needed for the image compose process and building the custom iso.

Then once the required packages are installed we need to enable the cockpit and osbuild-composer services.

$ sudo systemctl enable --now osbuild-composer.socketCreated symlink /etc/systemd/system/sockets.target.wants/osbuild-composer.socket → /usr/lib/systemd/system/osbuild-composer.socket.

Now that we have our prerequisites let us move onto the image building process for building our Red Hat Device Edge Arm with MicroShift. Unlike the previous blog, where we synced down the additional repositories we needed, here we will just download the packages that we will need and create our own custom repository. The following packages will need to be manually downloaded and saved into the microshift-local repository directory we will create.

Now notice in the output above there are some additional packages we pulled with regards to openvswitch3 and python3. These packages were pulled because currently MicroShift requires openvswitch2.17 but that did not come with Red Hat Enterprise Linux 9.2 Beta, only openvswitch3 did. So what we did to work around that temporarily is use the openvswitch3 srpms to build an openvswitch2.17 rpm we could use for this demonstration. The python3 packages were requirements we needed to get the rpmbuild to compile appropriately. This issue is being addressed here.

Now we can use the createrepo command to create a local repository of those packages we just downloaded.

With the repository created we now need to build a repository toml (Tom's Obvious Minimal Language) that defines the packages source.

Take the toml file we created above and apply it to the osbuild-composer environment by adding it as a source. Once we have added it as a source we can validate what sources are available by listing them out.

$ sudo composer-cli sources listappstreambaseosmicroshift-local

Now that we have all the package sources setup for our Red Hat Device Edge MicroShift image we can now begin to construct the toml file that will define our image. In our toml we will define a version, the packages to be included in the image and what services to be enabled.

The blueprint toml we created above can now be pushed into the osbuild-composer and we can validate it is there by listing the available blueprints.

$ sudo composer-cli blueprints listrhde-microshift

Once the blueprint is pushed up we should be able to compose our image. However when dealing with the Red Hat Enterprise Linux 9.2 Beta I found that the default repos for ImageBuilder pointed to the standard location for 9.2 which have yet to be made available. This will cause an error when trying to compose:

We can however work around this issue by changing the following following file and updating the two content paths accordingly. Or alternately copy the original file into the following path: /etc/osbuild-composer/repositories/ and then make the edits. The latter being the preferred method though I am using the former here.

https://cdn.redhat.com/content/dist/rhel9/9.2/aarch64/baseos/os

to

https://cdn.redhat.com/content/beta/rhel9/9/aarch64/baseos/os

and

https://cdn.redhat.com/content/dist/rhel9/9.2/aarch64/appstream/os

to

https://cdn.redhat.com/content/beta/rhel9/9/aarch64/appstream/os

At this point we are now ready to compose our image by issuing the composer-cli compose command to start building our image. In our case the image will use the rhde-microshift blueprint and will build a rhel-edge-container.

The process of building the image can take some time depending on the system it is run on. We can watch the progress either by running the composer-cli compose status command over and over or place a watch in front of it.

$ watch sudo composer-cli compose status

Once the image has finished being built we should see a status like the one below.

Now we need to pull down a local copy of the image so we can work with it by using composer-cli compose image.

One the image file is downloaded we next need to copy it into the local container-storage of our host and tag it accordingly. We can also validate it is there by running a podman images command.

$ sudo podman imagesREPOSITORY TAG IMAGE ID CREATED SIZElocalhost/rhde-microshift latest 733a820bb1bf 4 hours ago 1.15 GB

Now we will go ahead and start the container locally with podman. We need to do this because we want to extract the contents of the container image.

$ sudo podman psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESb2877a9d741c localhost/rhde-microshift:latest nginx -c /etc/ngi... 9 seconds ago Up 9 seconds 0.0.0.0:8000->8080/tcp compassionate_matsumoto

With our Red Hat Device Edge container image running we now need to create a directory structure that will be the location for the artifacts we need to gather so we can generate a complete zero touch Red Hat Device Edge bootable iso image. First we will create the generate-iso directory and an ostree subdirectory inside. We will copy the repo directory from the running container into this ostree subdirectory. Once we have completed the copy we can stop the container as it will no longer be needed. We can also validate the contents of the ostree/repo directory to confirm it looks like the listing below.

$ sudo podman cp b2877a9d741c:/usr/share/nginx/html/repo ~/generate-iso/ostree

$ sudo podman stop b2877a9d741cb2877a9d741c

$ sudo ls -l ~/generate-iso/ostree/repototal 16-rw-r--r--. 1 root root 38 Apr 7 14:15 configdrwxr-xr-x. 2 root root 6 Apr 7 14:15 extensionsdrwxr-xr-x. 258 root root 8192 Apr 7 14:15 objectsdrwxr-xr-x. 5 root root 49 Apr 7 14:15 refsdrwxr-xr-x. 2 root root 6 Apr 7 14:15 statedrwxr-xr-x. 3 root root 19 Apr 7 14:15 tmp

Now that we have our Arm based rpm-ostree image staged we can move onto creating a few additional artifacts we need for our zero touch boot iso. The first one we need is the grub.cfg:

For our zero touch provisioning workflow we also need a kickstart file to automate the installation process. The kickstart below is a straight forward example however I want to point out a few things that of interest

$ chmod 755 ~/generate-iso/recook.sh

Let's now confirm that our directory structure looks correct. We should have two config files, a script, our ostree directory with the image contents in it and Red Hat Enterprise Linux 9.2 Beta source iso.

At this point if everything looks good from the directory structure layout we should now be able to generate our zero touch Red Hat Device Edge iso using the recook script we created in a few steps above.

Drive current: -outdev 'stdio:../rhde-ztp.iso'Media current: stdio file, overwriteableMedia status : is blankMedia summary: 0 sessions, 0 data blocks, 0 data, 45.4g freexorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA 119 rulesxorriso : NOTE : -as mkisofs: Ignored option '-cache-inodes'xorriso : UPDATE : 28600 files added in 1 secondsAdded to ISO image: directory '/'='/tmp/tmp.yB4mEW6FUz/new/iso'xorriso : UPDATE : 30833 files added in 1 secondsxorriso : UPDATE : 30833 files added in 1 secondsxorriso : UPDATE : 1.86% donexorriso : UPDATE : 32.56% donexorriso : UPDATE : 52.86% donexorriso : UPDATE : 62.29% done, estimate finish Sun Apr 09 10:35:29 2023xorriso : UPDATE : 72.98% done, estimate finish Sun Apr 09 10:35:30 2023xorriso : UPDATE : 90.36% doneISO image produced: 898514 sectorsWritten to medium : 898514 sectors at LBA 0Writing to 'stdio:../rhde-ztp.iso' completed successfully.

+ implantisomd5 ../rhde-ztp.isoInserting md5sum into iso image...md5 = ff42112cd2e7501c7ca21affa9a1b261Inserting fragment md5sums into iso image...fragmd5 = bcd4bb8223e1162623e881a2c8548de7cfffea173483e8d9a48795365d88frags = 20Setting supported flag to 0+ popd~/generate-iso+ mv /tmp/tmp.yB4mEW6FUz/new/rhde-ztp.iso ./+ rm -rf /tmp/tmp.yB4mEW6FUz++ stat -c %U:%G .+ chown bschmaus:bschmaus ./rhde-ztp.iso

Once the script is completed we should have a rhde-ztp.iso in our directory.

Take the iso and either write it onto a usb drive or copy it to a hypervisor where the Arm virtual machine can consume it. I am doing the latter for this demonstration. In the previous blog we showed a video of the device booting up and the kickstart configuration doing the heavy lifting. Since we would be seeing the same thing in the previous video on this time it is on Arm I will defer the video of that process.

Once the edge virtual machine has rebooted we should be able to login into the host and confirm MicroShift is fully operational.

$ cat /etc/redhat-releaseRed Hat Enterprise Linux release 9.2 Beta (Plow)$ uname -aLinux adlink-vm3.schmaustech.com 5.14.0-283.el9.aarch64 #1 SMP PREEMPT_DYNAMIC Thu Feb 23 19:37:21 EST 2023 aarch64 aarch64 aarch64 GNU/Linux

$ oc get all -ANAMESPACE NAME READY STATUS RESTARTS AGEopenshift-dns pod/dns-default-rflw5 2/2 Running 3 38hopenshift-dns pod/node-resolver-95lcl 1/1 Running 1 38hopenshift-ingress pod/router-default-64fc9949cd-tbj2d 1/1 Running 2 38hopenshift-ovn-kubernetes pod/ovnkube-master-ppwj8 4/4 Running 7 38hopenshift-ovn-kubernetes pod/ovnkube-node-zbnhd 1/1 Running 3 (16m ago) 38hopenshift-service-ca pod/service-ca-67df7c6965-bzv4v 1/1 Running 1 38hopenshift-storage pod/topolvm-controller-59974b64d9-thj8z 4/4 Running 4 38hopenshift-storage pod/topolvm-node-w9kp8 4/4 Running 9 (16m ago) 38h

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEdefault service/kubernetes ClusterIP 10.43.0.1 443/TCP 38hopenshift-dns service/dns-default ClusterIP 10.43.0.10 53/UDP,53/TCP,9154/TCP 38hopenshift-ingress service/router-internal-default ClusterIP 10.43.212.134 80/TCP,443/TCP,1936/TCP 38h

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGEopenshift-dns daemonset.apps/dns-default 1 1 1 1 1 kubernetes.io/os=linux 38hopenshift-dns daemonset.apps/node-resolver 1 1 1 1 1 kubernetes.io/os=linux 38hopenshift-ovn-kubernetes daemonset.apps/ovnkube-master 1 1 1 1 1 kubernetes.io/os=linux 38hopenshift-ovn-kubernetes daemonset.apps/ovnkube-node 1 1 1 1 1 kubernetes.io/os=linux 38hopenshift-storage daemonset.apps/topolvm-node 1 1 1 1 1 38h

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGEopenshift-ingress deployment.apps/router-default 1/1 1 1 38hopenshift-service-ca deployment.apps/service-ca 1/1 1 1 38hopenshift-storage deployment.apps/topolvm-controller 1/1 1 1 38h

NAMESPACE NAME DESIRED CURRENT READY AGEopenshift-ingress replicaset.apps/router-default-64fc9949cd 1 1 1 38hopenshift-service-ca replicaset.apps/service-ca-67df7c6965 1 1 1 38hopenshift-storage replicaset.apps/topolvm-controller-59974b64d9 1 1 1 38h

We have confirmed MicroShift is fully functional here and ready to deploy workloads. Hopefully this blog provides an idea of what the workflow process looks like with Red Hat Device Edge on Arm with Red Hat Enterprise Linux 9.2 and MicroShift 4.13. The process looks fairly similar to the x86 workflow but there are a few nuances as pointed out in this blog when dealing with Arm.